All Questions
Tagged with python-3.xweb-scraping
206 questions
5votes
2answers
661views
Readability and error handling improvements for Python web scraping class
Description I recently wrote a Python script to download files from the Library of Congress (LOC) based on a search query. The code fetches metadata, extracts file ...
- 253
3votes
1answer
86views
Scraping website with Python and Selenium to collect data from dynamic website
Summary: The code scrapes the website and collects the data to store it in CSV. It also downloads selected information that is available for download in PDF format. The details and the entire code are ...
- 269
2votes
1answer
66views
A selenium web scraper to package NBA data
I'm building a selenium web scraper for basketball-reference.com that takes a player name and returns data in either a JSON format or Pandas DataFrame object. The class in question is one of many that ...
1vote
0answers
75views
Search Stack Overflow and GitHub for code in a specified language
This code is designed to scrape Stack Overflow and GitHub, pulling information based on a user-specified programming language and processing the data into a format for AI learning. It uses a number of ...
3votes
1answer
220views
A simple web scraper for nature.com news articles
I have created a simple web scraper that fetches news article previews from nature.com and saves each article to a file containing the article preview text. I am learning independently, so I would ...
3votes
2answers
226views
Saving Scraped Data to a File
When scraping and saving data into a file, Which method is more efficient when saving scraped data to a file? open the file first, scrape, and save the data all ...
- 191
3votes
1answer
53views
request data and print results
On last test, the below code takes approximately 10 seconds to download then print the data from 10 url's. I wish to speed this up as much as possible as later on I plan to expand this further and use ...
- 67
3votes
1answer
200views
Scrape PokeDex and display in tkinter
Hi I am new here and I just completed my first working version of a pokedex app with a GUI using tkinter. I used selenium to scrape the data from pokemondb.net, and then used pandas to clean up the ...
9votes
2answers
2kviews
Python script to scrape and parse the Stanford Encyclopedia of Philosophy
I wrote the following script to parse an SEP article and call pandoc to convert it to EPUB. I'd love your feedback. There is no function but I didn't think it was worth adding. Also there is no test ...
user260644
1vote
0answers
66views
How to make memory and cpu efficient a code
I am using a python scraper code to grab publicly available data from http://103.48.16.132/echalan/ but it takes almost ~6gb of memory and more cpu. I need to run ...
- 111
3votes
0answers
802views
A simple Python script that crawls information about Youtube playlists and your watch history
You will need to follow this guide. This is a Youtube crawler that crawls information about Youtube playlists, it uses Youtube Data API v3 and it crawls the title, url, description, count and videos ...
- 3,406
3votes
0answers
88views
Scraping housing dataset row by row from Estate Agent's website
I have written web scraping code with selenium with python3 for a dynamic site (it was my first project that is why I started with selenium). In my code I extracted the values and created DataFrame. ...
8votes
1answer
148views
Scrape a URL for articles to save
Below is an exercise project I was doing on a certain educational site. It is supposed to parse a given (static address in this example) URL for html data, search articles of a given type there and ...
4votes
2answers
603views
Web scraping data.cdc.gov for COVID-19 Data with Selenium in Python
I'm attempting to scrape data.cdc.gov for their COVID-19 information on cases and deaths. The problem that I'm having is that the code seems to be very inefficient. It takes an extremely long time for ...
- 55
2votes
0answers
165views
Web scraping articles using asyncio
I use asyncio to speed up web scraping. I collect only title, author, tags, datetime, total comments from list view from specific website. Also, i collect these from all pages. I would like to improve ...
- 417
